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Explanation of State Profiles 



T he relations between the National Assessment of Educational Progress 
(NAEP) results and individual state assessment results vary from state to state. 
Individual state profiles in this section display the comparisons for each state. 
Each state profile has up to 13 elements, depending on the availability of school' level 
state assessment information in the national longitudinal school-level state 
assessment score database (NLSLSASD). They include: 

• a summary description of the state assessment data; 

• an overview of the results displayed in the profile; 

• a display of the state’s achievement standard thresholds on the NAEP 
achievement distribution in the state; 

• the correlations between NAEP and state assessment school achievement; 

• the percentages of students with disabilities or English language learners; 

• a comparison of NAEP and state assessment achievement changes; 

• state-reported percentages of students meeting standards; and 

• comparisons of NAEP and state assessment achievement gaps. 

These are described below, in the context of the example profile displayed on the 
following pages. 
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Element 1 

Brief description of the state assessment data 

The description is based primarily on information provided on the state education 
agency website, as it applies to the data used in this comparison report (schooblevel 
scores on reading and mathematics assessments). The information included in the 
descriptions includes test(s) used, grades tested, subgroup data availability, 
availability of data across years, and data suppression information, as well as any 
information which would be required for understanding the results presented in the 
profile. 

Element 2 

Brief textual summary of statistically significant differences 
between NAEP and state assessment scores. 

The summary provides a brief overview of the results being displayed in the profile. It 
includes the number of schools in each grade which are being used for the 
comparison, a textual explanation of the standards comparison graphs (element 3), a 
brief explanation of the changes in achievement (element 6), and a summary of 
significant results for each gap type (Black' White, Hispanic- White, and poverty- 
elements 8-13). 1 The summary serves to highlight the information presented in the 
graphs and tables. 



1 . The poverty gap in achievement refers to the difference in achievement between economically 
disadvantaged students and other students, where disadvantaged students are defined as those eligible 
for free/reduced price lunch. 
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Figure D-1 . Elements 1 and 2 of the state profile 




State X 




T hrough the Comprehensive Assessment System, X administers exams in 
grades 4 and 7 in English language arts and grades 4 and 8 in mathematics. 
Scores are available for Hispanic and Black students, but there are too few 
Black students in grade 8 to provide a reliable comparison. State X uses four 
achievement levels for reporting purposes: warning, needs improvement, proficient, and 
advanced. Assessment scores based on 9 or fewer students are suppressed. 




Summary of Comparisons 

The results of comparisons between NAEP and state assessment results, which for 
2003 are based on 189 schools in grade 4 and 145 schools in grade 8, are shown 
graphically on the following pages. A brief summary of the results follows: 1 



• Standards. The state’s primary grade 4 mathematics performance standard 
(proficient) is close to the NAEP proficient level. This is also true for grade 8. 

• Trends. Between 2000 and 2003, NAEP reported a gain in grade 4 in percent 
proficient, which the state did not. Between 2000 and 2003, the NAEP grade 8 
gains in percent proficient are greater than the state assessment gains. 

• Gaps. Overall, the Black- White gap in grade 4 in percent meeting the state’s 
standard in mathematics in 2003 was greater when measured by NAEP compared 
to the state assessment. There were insufficient data for comparing the NAEP 
and state assessment measurement of the Black- White gap in mathematics in 
grade 8 in 2003. Overall, there were no significant differences between NAEP 
and the state assessment in measurement of the Hispanic-White gap in 
mathematics in grades 4 and 8 in 2003. There were insufficient data for 
comparing the NAEP and state assessment measurement of the poverty gap in 
mathematics in grades 4 and 8 in 2003. 



1. All statements of differences are based on statistical tests at the 5% significance level. However, 
these results must be considered in the context of the available data. NAEP and state assessments 
may employ different test items, testing accommodations, and scoring methods; they may involve 
different students in each school, at different times of the year, with different motivational 
characteristics. At the present time, in spite of controlling for effects of school sampling, differences 
in standards, and NAEP exclusion rates, we cannot identify specific reasons for differences between 
NAEP and state assessment results. 
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Element 3 

Position of standards in the achievement distribution 

The position of the state’s achievement standard thresholds on the NAEP 
achievement distribution in the state are based on mapping the percentages 
achieving state standards reported for schools participating in NAEP with the 
distribution of NAEP grade 4 or grade 8 performance in those schools. In some cases, 
the state’s standard is for an adjacent grade. In those cases, the assumption is made 
that the percentage of students meeting the state’s standard for one grade would be 
approximately the same as the percentage meeting a standard the state might set for 
the next grade. The distributions are displayed for all states with available 
percentages achieving standards in NAEP schools. 

Because Alabama, Tennessee, and Utah data files available for this report do not 
include percentages of students meeting standards, the state profiles for these states, 
unlike the other states, are based on the median percentile rank in each school, not 
the percentages meeting state standards. Therefore, no state standard thresholds are 
placed on the NAEP scale. 

Element 4 

Correlations of NAEP and state assessment school achievement 

Based on schools participating in NAEP, this table displays correlations of 
percentages reported as meeting state standards with NAEP percentages of 
achievement meeting the estimated state standard in the same schools. For this 
display, NAEP has been rescored to estimate the percentages of students above the 
state’s cutpoints indicated in element 3. 

In states with multiple standards, one standard was identified for this report as the 
primary standard. In nearly every case, this is the standard that is used for reporting 
adequate yearly progress to the federal government. For Alabama, Tennessee, and 
Utah, the correlations are for median percentile ranks. 



2. The figure plots the relative frequency of the NAEP plausible values in the state. Since the numerical 
values on the vertical axis (i.e., the relative frequencies, or more accurately, approximate probability 
densities) are solely a function of the fineness of the categorization of the continuous scale on the 
horizontal axis, it is neither meaningful nor appropriate to display numerical values for the vertical 
axis. 
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Figure D-2. Elements 3 and 4 of the state profile 




Figure 1. Distribution of grades 4 and 8 NAEP mathematics achievement scores: 
2003 
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SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statis- 
tics, National Assessment of Educational Progress (NAEP), 2003 Mathematics Assessment: Full population esti- 
mates. The National Longitudinal School-Level State Assessment Score Database (NLSLSASD) 2004. 



Table 1. School-level correlations between NAEP and state assessment of 
percentages of students achieving state's mathematics standards: 2003 



Standard 


Grade 4 




Grade 8 


Correlation Standard error 


Correlation 


Standard error 


Needs Improvement 


0.78 


0.015 


0.88 


0.015 


Proficient 


0.82 


0.008 


0.87 


0.012 


Advanced 


0.74 


0.033 


0.87 


0.023 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statis- 
tics, National Assessment of Educational Progress (NAEP), 2003 Mathematics Assessment: Full population esti- 
mates. The National Longitudinal School-Level State Assessment Score Database (NLSLSASD) 2004. 




D-2 l National Assessment of Educational Progress 



Comparison between NAEP and State Mathematics Assessment Results: 2003 




Element 5 

Percentages of students with disabilities or English language 
learners 

Because measurement of trends in achievement can be affected by changes in the 
percentages of students with disabilities (SD) or English language learners (ELLs), 
through their exclusion from testing or access to testing accommodations, information 
about these percentages are presented for NAEP assessments in 2000 and 2003. The 
percentages are presented separately for ( 1 ) English language learners (but not with a 
disability), (2) students with disabilities (who are not English language learners), and 
(3) English language learners who also have a disability. The percentages of students 
identified with disabilities or as English language learners who participated in NAEP 
without accommodations are not included in the table. 

The percentages of students excluded from NAEP participation are based on the total 
student population. For example, if 10 percent of students have a disability and 40 
percent of those with a disability are excluded, that means that 4 percent of the total 
student population is excluded. The use of full population estimates in this report is 
intended to minimize the effects of NAEP exclusions on the results of changes in 
achievement. Similarly, the percentages of students accommodated by NAEP are based 
on the total student population. In the example above, if 50 percent of the included 
SD/ELL students were accommodated, that would mean that accommodations were 
provided for 50 percent of the included 6 percent, or 3 percent of the total population. 

Element 6 

Comparison of NAEP and state assessment changes in 
achievement, based on NAEP schools 

Achievement changes are presented as percentages meeting the states’ standards in 
NAEP schools for state assessment results (lighter line) and for NAEP results (darker 
line). The standards are equated in the first year of analysis, forcing the percentages to 
match in the first year by definition. Differences between NAEP and state assessment 
achievement changes are revealed at the second point in time. Asterisks on the charts 
indicate statistically significant differences (p<.05) between NAEP and state 
assessment gains. 

Comparisons of achievement changes are available only for states in which comparable 
state scores are reported across years. Many states changed tests or changed standards 
between 2000 and 2003 and, although data were available for the different tests, it is 
impossible to construct meaningful comparisons of NAEP and state assessment gains. 

Element 7 

State reported percentages meeting standard 

The changes in achievement presented in element 6 are based on the NAEP sample of 
schools, weighted to represent the state. In most states, these results can be compared 
to reports issued by state education agencies on their websites. 3 These are shown in 



3. The state-reported percentages were retrieved from state education agencies’ websites in July 2004. 
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element 7. Ideally, the percentages in the table of state-reported achievement should 
match the state assessment percentages based on the NAEP sample of schools. 
However, in some cases state assessment scores were not available for all NAEP 
schools. This occurs, for example, when state assessment scores are for an adjacent 
grade and some NAEP schools do not include the grade tested, or when they have not 
been reported by the state. 



Figure D-3. Elements 5, 6, and 7 of the state profile 



STATE X 





Table 2. Percentages of English language learners and students with disabilities 
identified, excluded, and accommodated in the NAEP mathematics 
assessments, by grade: 2000 and 2003 





Grade 4 




Grade 8 




Students 


2000 


2003 


2000 


2003 


Identified 


19.4 


21.9 


19.4 


18.4 


English language learner 


5.1 


3.8 


3.0 


2.0 


Student with disability 


13.7 


17.0 


15.6 


15.2 


Both 


0.6 


1.0 


0.9 


1.2 


Excluded 


2.7 


2.9 


2.7 


3.1 


English language learner 


2.0 


0.8 


0.9 


0.8 


Student with disability 


0.7 


1.8 


1.2 


1.8 


Both 


0.0 


0.3 


0.6 


0.5 


Accommodated 


10.1 


15.0 


8.8 


10.8 


English language learner 


1.5 


1.1 


1.1 


0.4 


Student with disability 


8.2 


13.3 


7.5 


10.0 


Both 


0.5 


0.6 


0.2 


0.5 



SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statis- 
tics, National Assessment of Educational Progress (NAEP), 2000 and 2003 Mathematics Assessments. 



Figure 2. Comparison of NAEP and state assessment achievement changes in 
percent meeting mathematics standards, by grade: 2000 and 2003 



grade 4 grade 8 




2000 2003 2000 2003 

Year Year 

* NAEP and state assessment 2000-2003 changes are significantly different (p <.05). 

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statis- 
tics, National Assessment of Educational Progress (NAEP), 2003 Mathematics Assessment: Full population esti- 
mates. The National Longitudinal School-Level State Assessment Score Database (NLSLSASD) 2004. 




Table 3. Percentage meeting mathematics standards as reported by state: 2003 



Level 


2000 


2003 


Grade 4 


40.0 


40.0 


Grade 8 


34.0 


37.0 



SOURCE: State education agency website. 
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